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Abstract 
We show how generalized least-squares fitting, namely, the fitting of correlated data, can be 
carried out with algorithms for uncorrelated data of constant variance. Doing so requires 
Article Info only a simple linear transformation of the measurement data. 
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1. Introduction 


Least-squares fitting of data is a recurring task in statistics and natural science. While there are many approaches and 
algorithms for performing least-squares fitting of uncorrelated data, only very few exist that support least-squares 
fitting correlated measurement data. This is not a major problem though, since—as we show in this application note— 
least-squares fitting of correlated measurement data can be translated into a least-squares fitting problem for uncorrelated 
data of constant variance. The only thing needed is a simple linear transformation of the data. 


2. The Optimization Problem 


Least-squares fitting can be described as follows. The relationship between the varaibles x,, x,, ... x,, is described by a 
model function. 


x= f(x, 0) .. (1) 


where x is a column vector of the x-values and ais the vector of the model parameters. An example for fis 


x} x 
X=| xX = Xx: 
2 2 (2) 
x3 ao +A)X, +A7Xy 


where fhas two input variables x, and x,, and one output variable x,. This example is illustrated in Figure 1. 
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In this application note, we assume that the x-values are complex. However, our findings also hold for 
real-valued x's. 


X 
/ model function f with ‘ 


Xx 
2 model parameters a 2 
X,= O,,tO,X, FOX, 


Figure 1: Illustration of the Model Function in Equation (2) 


The problem to be solved is that the true x- and @- values are unknown. For instance, the x values can only be 
observed with additive noise. Generalized least-squares curve fitting provides a way of estimating the true values. The 


estimates are written as {and @ and they are inferred by minimizing Equation (3). 


=6'C'6 .(3) 
where 
6 :=x— f (%,a) AA) 


The elements of the matrix C are defined as: 

C, = E({x, -e() Hal - EGY) ...(5) 
for 1 <i, j <m [see, for instance (Taupin, 1988)]. estands for the expectation value. The minimization of Equation (3) is 
referred to as generalized least-squares fitting. 


Note that the x-values in Equation (5) are modeled as stochastic variables. For more details on observed values and 
stochastic variables see Appendix A. 


Also note that C,, is the variance of x, and that C,, for i # j indicates the correlation between x, and x;. Furthermore, note 
that if fis non-linear in x, fmay have to be replaced with a slightly modified function. For more details on this see (Metz 
etal., 2003). 


While many analytical formulas and numerical procedures exist for the minimization of 7 for uncorrelated x-values 
(C,,=0 for i #7), few exist for correlated x-values. This is especially true for commercially available procedures, which 
only seem to tackle the minimization of 7’ for uncorrelated x values with constant variance. In other words, they only 


~=5'5 ..(6) 
The minimization of Equation (6) is referred to as ordinary least-squares fitting. 
The next section discusses how generalised least-squares fitting problems can be transformed into ordinary 
least-squares fitting problems. 
3. Translating Generalized Least-Squares Fitting into Ordinary Least-Squares Fitting 


This transformation exploits the fact that C is hermitian [namely Ct = C according to Equation (5)]. See also (Wikipedia 
Contributors, 2021). Therefore, C can be factored into: 


Cc=UVU 7) 
where U and Vare constructed as follows (Wikipedia Contributors, 2020): solve the eignenvalue equation 


Cu=vu ...(8) 
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The solutions are the vectors uw, u,,...u, and the real values v,, v, ... v,. Thereafter, construct U according to 


) 


and V according to 


U=(U,, U,,...U 


m 


_ |v; for i= j 
u” \0 for i# j 


Note that Uis a unitary matrix, meaning U* = U' (Wikipedia Contributors, 2020). 


0=U0oS§' 
with 
av; for i=j 
oy = ae 
O for i+ j 


transforms Equation (3) into Equation (6). To demonstrate this, we insert Equation (11) into Equation (3): 


L=6'C'6 
= (Uo §')'C'(Ua 5’) 
=(8')io'U'C'Uos' 
Using Equation (7), this yields’ 
H=(6' oU(UVW'y'Uos' 


= nt 40,7) 5,-1 ' 
=(6')'ou'u'y'v Ea 
1 


=(5'JioVia6' 
where | is the identity matrix. 


Remembering that V=oo [see Equation (12)], we get 


1° = (6! go"!e'08' 
1 1 


=(6')'5' 
which indeed is isomorphic to Equation (6) (Wikipedia Contributors, 2021). 


4. How to Use this Result in Practice? 
From Equation (11) and Equation (4), we get 
6! — o!U"'6 
=o 'U'x-o'U' f(%,4) 
= gus ~o 'U'f (8,4) 
# f'G@) 


Aron 


The last row exploits the fact that 6’ = x'— f'(x',a). 


9) 


(10) 


ai) 


(12) 


mie 


(14) 


(15) 


...(16) 


From Equation (16), the following recipe for fitting correlated data with algorithms for ordinary least-squares fitting 


can be inferred. 


1. Take your measurement data x and calculate 


x'=0 'U'x 


(17) 


Joachim W. Walewski and Thomas M ez / Int.J.Data.Sci. & Big DataAnal. 1(3) (2021) 1-5 Page 4 of 5 


2. Calculate a new model function 


f'(x,a)=0'U' fUo %',a) (18) 
3, Minimize 

(6')'5" (19) 
with 

5'=x'- f'(x,a) (20) 


This step yields the estimates ~’ and ¢ 


4. Calculate the estimates * with 


R=Uo # (21) 


5. Conclusion 

This application note provides a simple recipe for how to translate generalized least-squares curve fitting into least- 
squares curve fitting of uncorrelated data with constant variance. 
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Appendix 


Stochastic Variables and Observations 


The reader should be aware of the distinction between stochastic variables and observations. Stochastic variables 
are a mathematical concept for representing quantities that are subject to random variation. Stochastic variables are 
associated with a probability law, a so-called distribution function, and one can assign expectation values, variances, 
co-variances and so on to them. 


Observations, on the other hand, are concrete numbers that are observed in, for instance, measurements, and 
they are typed into, say, a computer when performing some kind of evaluation. 


Many textbooks on statistical inference use capital letters such as X for stochastic variables and small letters 
such as x for observations. In this work, we do not make this formal distinction in order to keep the formalism simple. 


From the context, it will be clear what is meant. For instance, expectation values, variances etc. are only defined 


for stochastic variables, whereas a minimization of Equation (6) can only be done for concrete numbers, namely, the 


observations x. 


Statistical inference is about estimating the values of unknown quantities. These estimates are obtained from the 
data of an experiment, so they are numbers you calculate from the observations. 
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